Robotic arm and control system

ABSTRACT

A robotic arm and control system includes a robotic arm which moves in response to one or more command signals. One or more “active” fiducials are located on the arm, each of which emits its own light. A 3D camera having an associated field-of-view is positioned such that at least one fiducial and a target object to be manipulated are in the FOV. To determine their spatial positions, the arm fiducials are activated and the target object is preferably illuminated with a scanning laser; the camera produces output signals which vary with the spatial locations of the fiducials and target object. A controller receives the output signals and uses the spatial position information as feedback to continuously guide the arm towards the target object. Multiple active fiducials may be employed, each having respective characteristics with which they can be differentiated.

RELATED APPLICATIONS

This application claims the benefit of provisional patent applicationNo. 60/966,137 to Robert L. Kay, filed Aug. 24, 2007.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to robotics, and more particularly torobotic arms and systems for controlling such arms.

2. Description of the Related Art

Robotic arms—i.e., articulated mechanisms designed to perform one ormore tasks, typically under the control of a computer—are commonly usedin factory and assembly line environments. For example, one common taskperformed by a robotic arm is to grip a target object and then move itto a desired location. This is accomplished with the help of an‘effector’ mechanism, typically located on the end of a robotic arm; atypical effector is a “gripper” which has at least one movable jaw thatis closed in order to grip or grasp a target object.

Traditionally, angle sensors such as encoders are incorporated into arobotic arm's joints, and are used to determine the position of the arm.By knowing the exact length of each portion of the arm, and the size andangle of each joint, a computer can calculate the spatial position ofthe arm.

However, there are drawbacks associated with this method of operation.For example, the accuracy with which an arm can be positioned may bedegraded as the mechanism's parts become worn. Positional accuracyrequirements may also limit the speed with which the robotic arm can bemanipulated, and require that a target object be precisely positionedprior to its being manipulated.

SUMMARY OF THE INVENTION

A robotic arm and control system are presented which overcome theproblems noted above, providing efficient and accurate effectorpositioning and movement under a variety of operating conditions.

A robotic arm in accordance with the present invention moves in responseto one or more command signals. One or more fiducials are located on thearm. Each fiducial is an “active” fiducial, meaning that each emits itsown light. A 3D sensing device having an associated field-of-view (FOV)is positioned such that at least one of the fiducials and a targetobject to be manipulated with the arm are in the FOV. To determine theirspatial positions, the arm fiducials are activated and the target objectis preferably illuminated with a scanning laser, thereby enabling thesensing device to produce output signals which vary with the spatiallocations of the fiducials and target object.

The system also includes a controller which receives the output signalsfrom the sensing device. The controller uses the spatial positioninformation provided by the sensing device as feedback to continuouslyguide the arm towards the target object. Providing a closed loop 3Dmotion control system in this way enables the arm to be made ofnon-precision construction, having few, if any, position and/or anglesensors.

Multiple active fiducials may be employed, each having respectivecharacteristics with which they can be differentiated. Active fiducialsmight also be deployed on the target objects, or on a holding fixture inwhich target objects are placed. Variations involving multiple roboticarms and/or multiple 3D sensing devices are also described.

These and other features, aspects, and advantages of the presentinvention will become better understood with reference to the followingdrawings, description, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating the basic configuration of a roboticarm and control system per the present invention.

FIG. 2 is a block diagram illustrating one possible embodiment of acontrol loop as might be used with a robotic arm and control system perthe present invention.

FIG. 3 is a block diagram illustrating additional details of a controlloop such as that shown in FIG. 2.

FIG. 4 is a diagram illustrating a configuration of a robotic arm andcontrol system per the present invention which includes a target objectholding fixture.

FIG. 5 is a diagram illustrating a configuration of a robotic arm andcontrol system per the present invention which includes two roboticarms.

FIG. 6 is a diagram illustrating a configuration of a robotic arm andcontrol system per the present invention which includes a first cameradedicated to imaging the system's robotic arm and a second cameradedicated to imaging a target object.

DETAILED DESCRIPTION OF THE INVENTION

The basic configuration of a robotic arm and control system inaccordance with the present invention is shown in FIG. 1. The systemincludes a robotic arm 10 which moves in response to one or more commandsignals 12. There is at least one “active” fiducial 14 located onrobotic arm 10. As used herein, an “active” fiducial is one that emitsits own light—i.e., it is self-illuminating.

The system also includes a 3D sensing device 16 having an associatedfield-of-view (FOV) 18, positioned such that at least one of thefiducials 14 on arm 10, and a target object 20 to be manipulated withthe arm, are in the sensing device's FOV. 3D sensing device 16 ispreferably a 3D camera, though other types of devices, such as [??],could also be used. However, for simplicity, sensing device 16 isreferred to as a 3D camera hereinafter.

The fiducials are preferably controllable—i.e., they can be activatedand deactivated on demand. To determine the spatial positions of the armand target object, the arm fiducials are activated and the target objectis preferably illuminated with a scanning laser, so that each can bedetected by 3D camera 16. 3D camera 16 is arranged to produce outputsignals 22 with which target object 20 and the spatial locations of thefiducials in FOV 18 can be determined.

A controller 24 receives output signals 22 from 3D camera 16, andprovides command signals 12 to robotic arm 10. Controller 24 is arrangedto provide closed loop control of arm 10: the controller uses thespatial position data provided by 3D camera 16 as feedback tocontinuously guide the arm towards target object 20. Providing a closedloop 3D motion control system in this way enables arm 10 to be made ofnon-precision construction, having few, if any, position and/or anglesensors.

Robotic arm 10 would typically include an effector 26, such as agripper, with which target object 20 is manipulated. To ensure that thespatial location of the gripper can be determined by the system, atleast one of the active fiducials is located on or a known distance fromthe effector.

As noted above, each active fiducial emits its own light, which ispreferably controllable so that the light can be commanded ON or OFF.When multiple active fiducials are employed within the FOV of the 3Dcamera, the fiducials have respective characteristics with which theycan be differentiated. For example, different fiducials can emit lighthaving different wavelengths (preferably within the visible spectrum), agiven fiducial could emit light which includes at least two differentwavelengths, a fiducial could be arranged such that the wavelength ofits emitted light changes in a predetermined pattern, or the emittedlight could be modulated in a predetermined pattern. Another possibleway to distinguish between fiducials is to have the fiducials formrespective predetermined patterns, such as a cross, a bar, or anequivalent feature. These differentiation techniques allow one or more3D cameras to operate and control one or more robotic arms insimultaneous operation. They also increase the 3D camera's detectioncapability when the background or environment contains other lightsources that might interfere with the identification process.

The system preferably employs multiple active fiducials on each roboticarm. This allows the 3D camera to lose sight of one or more fiducials,but still have at least one fiducial visible with which spatial positioninformation can be determined.

As noted above, each active fiducial emits its own light. This can beachieved by using light-emitting diodes, incandescent lamps, and/oroptical fibers, for example, or any other compact light sources that maybe mounted on the robotic arm or effector.

It may be possible for 3D camera 16 to detect target object 20 anddetermine its spatial location as long as the object is within the FOVof device 16. However, the system preferably includes a means 30 ofilluminating target object 20, to improve the ability of 3D camera 16 todetermine the spatial location of the target object.

One possible illumination means 30 is a scanning laser, which uses alaser to scan the target object and thereby enable 3D camera 16 todetect the target object and ascertain its spatial location. Such alaser would preferably be incorporated into 3D camera 16. The laserilluminates target object 20 with a scanning spot or line. 3D camera 16,which preferably comprises dual 2D cameras separated by a knowndistance, triangulates on the laser spot or line to generate a 3D imageof the target object. Thus, by illuminating target object 20 andactivating one or more fiducials on arm 10, 3D camera 16 determines thespatial locations of both target object 20 and effector 26, from whichthe target object's location in space relative to robotic arm 10 can bedetermined. Use of a scanning laser could also enable the illuminatedtarget object to be identified and measured.

A diagram illustrating the closed loop operation of the present systemis shown in FIG. 2. In this exemplary embodiment, controller 24comprises image processing circuitry 32 and servo processing circuitry34. The functionalities provided by controller 24 are preferablyimplemented digitally, using a software-driven processor, but might alsobe implemented with analog circuitry. Image processing circuitry 32receives signals 22 from 3D camera 16, and produces outputs 36 whichvary with the spatial positions of fiducials 14 and target object 20.Image processing circuitry 30 is also arranged to control the operationof 3D camera 16, fiducials 14 and target illuminator 30.

Servo processing circuitry 34 receives spatial position signals 36, andprovides command signals 12 to robotic arm 10 such that fiducials 14 aremoved towards target object 20; in this way, command signals 12 serve asfeedback to continuously guide arm 10 towards target object 20, therebyproviding closed loop 3D motion control.

One possible sequence of operations for a system which includes a 3Dcamera and laser scanner proceeds as follows:

1. Controller 24 prepares to move robotic arm 10 and its effector 26 toa spatial location to be determined.

2. Controller 24 enables scanning laser 30 to locate and illuminate oneor more target objects 20, which is imaged by 3D camera 16. When done,the controller disables the scanning laser and commands 3D camera 16 tooperate in a fiducial detection mode.

3. Controller 24 activates one or more of the active fiducials onrobotic arm 10. This may include initiating the characteristic behaviorswhich differentiate the fiducials as discussed above.

4. 3D camera 16 detects the active fiducial(s). The camera may perform asubtractive action by capturing “scene” data when the fiducials are offand then again when they are on, so as to remove the effects ofbackground or other interfering items or sources in the field of view.

5. The 3D camera (or computations performed in controller 24) computesthe spatial locations of the active fiducials and target object.

6. Controller 24 computes the difference between the spatial location oftarget object 20 and the spatial location of end effector 26. Using thisinformation, controller 24 produces command signals 12 which cause arm10 to be guided towards target object 20.

7. Steps 3 through 6 are executed until end effector 26 reaches thedesired location.

A diagram illustrating additional details of image processing circuitry32 and servo processing circuitry 34 is shown in FIG. 3. The location oftarget object 20 is preferably ascertained first, using a scanning laseras described above. The spatial positions of the active fiducials aredetermined next. Image processing circuitry 32 preferably includes abackground rejection processing module 40, which is arranged to performthe subtractive action described above in step 4. Here, 3D camera 16captures the scene in its FOV when the active fiducials are notilluminated, and again when they are illuminated. Background rejectionprocessing module 40 subtracts the non-illuminated image from theilluminated image, and outputs the result.

A fiducial detection and separation module 42 receives the imageproduced by background rejection processing module 40. Assuming thereare multiple fiducials in the FOV, and each has characteristics whichenable it to be differentiated from the others, fiducial detection andseparation module 42 detects the presence of each fiducial and notes itscharacteristics.

The characteristics of the respective fiducials are passed on to afiducial cross-referencing and arm position computation module 44, whichhas access to a library 46 that includes of information on each fiducialand its corresponding characteristics. By identifying and knowing thespatial position of each fiducial, and combining this data withknowledge of where each fiducial is physically located on robotic arm10, module 44 can compute the spatial location of the arm.

As noted above, image processing circuitry 32 is also arranged toactivate the scanning laser and determine the spatial location of targetobject 20. Data 36 representing the spatial locations of arm 10 andtarget object 20 is provided to servo processing circuitry 34.

Servo processing circuitry 34 accepts the spatial position data and,based upon the sensed position of the arm, determines the requiredcommand signals 12 to output to the robotic arm drive system. In apreferred implementation, the position data is operated on by servoerror and forward kinematics algorithms 48, followed by one or moreservo compensation algorithms 50 which allow the arm to performefficient, smooth motion based upon the characteristics of the arm. Asrobotic arm 10 moves, 3D camera 16 captures the positions of the variousfiducials, with image processing circuitry 32 controlling each of theactive fiducials and the 3D camera.

Note that the implementations of controller 24 described above aremerely exemplary. It is only essential that controller 24 be arranged toreceive spatial position data from a device such as a 3D camera, and toprovide closed loop 3D motion control of a robotic arm in response.

As noted above, 3D camera 16 preferably comprises dual 2D cameras. Thecameras are separated by a known distance, and, when a scanning laser isemployed, triangulate on the spot or line created by the scanning laserto generate a 3D image of the target object. The 2D cameras are suitablyCCD-type cameras. A laser and galvanometer are added to form a laserscanning 3D camera.

In many applications, it is not practical to place active fiducials onthe target objects. The present system avoids the need for suchfiducials by illuminating the target objects with a scanning laser orsimilar device, and then determining the position of the illuminatedobject with the 3D camera.

However, as an alternative to a scanning laser, active fiducials mightalso be located on the target object. This is illustrated in FIG. 4.Here, one or more active fiducials are positioned on target object 20.When so arranged, 3D camera 16 can determine the spatial location oftarget object 20 in the same way that it determines the spatial locationof robotic arm 10—by imaging the object while its fiducials areactivated—thereby eliminating the need for a scanning laser.

In some applications, a holding fixture such as a test tube holder orholding bin is employed to hold a target object while it is beingmanipulated by robotic arm 10. This possibility is also illustrated inFIG. 4. Here, target object 20 is placed within a holding fixture 62. Inthis case, one or more active fiducials 64 are preferably located on thefixture, so that its spatial location may be determined with 3D camera16. If it is necessary to pinpoint the location of the target objectitself, one or more active fiducials could also be located on the targetobject, or scanning laser 30 could be used for this purpose.

A system in accordance with the present invention could be implementedusing any of a number of different configurations. For example, as shownin FIG. 5, 3D camera 16 could be used to control robotic arm 10, and asecond arm 70. In this arrangement, arm 70 includes one or more activefiducials 72 which can be differentiated from those used on arm 10. Bothfiducials 72 and the target object 74 to be manipulated by arm 70 mustbe within the FOV of 3D camera 16, or the system must be arranged suchthat the position of 3D camera 16 can be moved as needed to accommodateboth arms. Controller 24 is arranged to process the spatial positioninformation for both arms, and provides command signals 12 and 76 toarms 10 and 70, respectively, to control their movements.

Another possible configuration is also shown in FIG. 5; here, roboticarm 70 is controlled with the use of a second 3D camera 80 and a secondcontroller 82. In this arrangement, fiducials 72 and target object 74are within the FOV 84 of camera 80.

Another possible configuration is illustrated in FIG. 6. Here, one 3Dcamera 90 is dedicated to imaging the active fiducials on robotic arm10, and a second 3D camera 92 is dedicated to imaging target object 20.Target object 20 can be sensed by any of the methods described above,including the use of a scanning laser or the use of an active fiducialon the object or its holding fixture. The imaging data from both camerasis fed to one or more controllers 94, which processes the data todetermine the spatial positions of arm 10 and target object 20, which isused as feedback to continuously guide the arm towards target object 20.Dedicated camera 90 might also be used to image active fiducials onother robotic arms, and dedicated camera 92 might also be used to imageother target objects.

The embodiments of the invention described herein are exemplary andnumerous modifications, variations and rearrangements can be readilyenvisioned to achieve substantially equivalent results, all of which areintended to be embraced within the spirit and scope of the invention asdefined in the appended claims.

1. A robotic arm and control system, comprising: a robotic arm whichmoves in response to one or more command signals; one or moreindependently controllable fiducials located on said arm, each of saidfiducials being an “active” self-illuminating fiducial such that eachemits its own light when activated; a 3D sensing device having anassociated field-of-view (FOV), positioned such that at least one ofsaid active fiducials and a target object to be manipulated with saidarm are in said sensing device's FOV, said sensing device arranged toproduce output signals with which the spatial locations of said at leastone of said active fiducials and said target object can be determined;and a controller arranged to control the activation of saidindependently controllable active fiducials, said controller furtherarranged to receive said output signals from said sensing device, toactivate one or more of said active fiducials, and to provide said oneor more command signals to said robotic arm, said controller arranged toprovide closed loop control of said arm such that said at least one ofsaid active fiducials is moved towards said object.
 2. The system ofclaim 1, wherein said robotic arm includes an effector with which saidtarget object is manipulated, at least one of said active fiducialslocated on or a known distance from said effector.
 3. The system ofclaim 1, wherein said active fiducials have respective characteristicswith which they can be differentiated.
 4. The system of claim 1, whereinone or more of said active fiducials emit light which includes at leasttwo different wavelengths.
 5. The system of claim 1, wherein one or moreof said active fiducials are arranged such that the wavelength of theiremitted light changes in a predetermined pattern.
 6. The system of claim1, wherein one or more of said active fiducials emit light that ismodulated in a predetermined pattern.
 7. The system of claim 1, whereinone or more of said active fiducials are arranged to form apredetermined pattern.
 8. The system of claim 1, wherein one or more ofsaid active fiducials comprise light-emitting diodes, incandescentlamps, and/or optical fibers.
 9. The system of claim 1, furthercomprising a means of illuminating said target object.
 10. The system ofclaim 9, wherein said means of illuminating said target object comprisesa scanning laser.
 11. The system of claim 1, wherein said 3D sensingdevice is a 3D camera, said controller arranged to: capture the image insaid camera's FOV when said active fiducials are not illuminated;capture the image in said camera's FOV when said active fiducials areilluminated; and subtract the non-illuminated image from the illuminatedimage and use the resulting image to determine the spatial locations ofsaid fiducials.
 12. The system of claim 1, wherein said controllercomprises: an image processing module arranged to: receive said 3Dsensing device's output signals; determine the spatial location of saidat least one of said fiducials on said robotic arm; and determine thespatial location of said target object; and a servo processing modulearranged to: receive spatial location data for said at least one of saidfiducials and said target object from said image processing circuit; andprovide said one or more command signals to said robotic arm such thatsaid controller provides closed loop control of said arm such that saidat least one of said fiducials is moved towards said target object. 13.The system of claim 12, wherein said image processing circuit is furtherarranged to control the operation of said at least one of said fiducialsand/or the positioning of said 3D sensing device.
 14. The system ofclaim 12, wherein said active fiducials comprise at least two fiducials,each of which is in said sensing device's FOV and emits light havingassociated characteristics which enables said fiducials to bedifferentiated, said image processing circuit further arranged to detectand determine the spatial positions of said at least two fiducials. 15.The system of claim 1, further comprising one or more active fiducialslocated on said target object, said 3D sensing device positioned suchthat at least one of said fiducials located on said target object is insaid sensing device's FOV, said sensing device arranged to produceoutput signals with which the spatial locations of said at least one ofsaid fiducials located on said target object can be determined.
 16. Thesystem of claim 1, wherein said target object is held in a fixture,further comprising one or more active fiducials located on said fixture,said 3D sensing device positioned such that at least one of saidfiducials located on said fixture is in said sensing device's FOV, saidsensing device arranged to produce output signals with which the spatiallocations of said at least one of said fiducials located on said fixturecan be determined.
 17. The system of claim 1, further comprising atleast one additional robotic arm, each of said additional arms moving inresponse to one or more command signals and having one or more activefiducials located on said arm.
 18. The system of claim 17, wherein said3D sensing device is positioned such that at least one of said fiducialson each of said arms is in said sensing device's FOV, said sensingdevice arranged to produce output signals with which the spatiallocations of said at least one of said fiducials on each of said armscan be determined.
 19. The system of claim 17, further comprising atleast one additional 3D sensing device, each of said sensing devicespositioned such that at least one fiducial on a respective arm and atarget object are in said sensing device's field-of-view (FOV).
 20. Thesystem of claim 1, wherein said 3D sensing device is a 3D camera. 21.The system of claim 1, wherein said controller is arranged such thatsaid active fiducials can be selectively turned on and off either one ata time or in groups.
 22. The system of claim 1, said system arrangedsuch that the spatial location of said target object is determinedwithout reference to any special markings or distinctive targets on saidtarget object.
 23. A robotic arm and control system, comprising: arobotic arm which moves in response to one or more command signals; oneor more independently controllable fiducials located on said arm, eachof said fiducials being an “active” self-illuminating fiducial such thateach emits its own light when activated; a first 3D sensing devicehaving an associated field-of-view (FOV), positioned such that at leastone of said active fiducials is in said first sensing device's FOV, saidfirst sensing device arranged to produce output signals with which thespatial location of said at least one of said active fiducials can bedetermined; a second 3D sensing device having an associated FOV,positioned such that at least one target object to be manipulated withsaid arm is in said second sensing device's FOV, said second sensingdevice arranged to produce output signals with which the spatiallocation of said target object can be determined; and a controllerarranged to control the activation of said independently controllableactive fiducials, said controller further arranged to receive saidoutput signals from said sensing devices, to activate one or more ofsaid active fiducials, and to provide said one or more command signalsto said robotic arm, said controller arranged to provide closed loopcontrol of said arm such that said at least one of said active fiducialsis moved towards said target object.
 24. A robotic arm and controlsystem, comprising: a robotic arm which moves in response to one or morecommand signals; one or more independently controllable fiducialslocated on said arm, each of said fiducials being an “active”self-illuminating fiducial such that each emits its own light whenactivated; an effector coupled to said arm with which a target object ismanipulated, at least one of said active fiducials located on or a knowndistance from said effector; a means of illuminating said target object;a 3D camera positioned such that at least one of said active fiducialsand said target object are in said camera's field-of-view (FOV), saidcamera arranged to produce output signals with which the spatiallocations of said at least one of said active fiducials and said targetobject can be determined; and a controller arranged to control theactivation of said independently controllable active fiducials, saidcontroller further arranged to receive said output signals from said 3Dcamera, to activate one or more of said active fiducials, and to providesaid one or more command signals to said robotic arm, said controllerarranged to provide closed loop control of said arm such that said atleast one of said active fiducials is moved towards said target object.25. The system of claim 24, wherein said means of illuminating saidtarget object comprises a scanning laser.
 26. A method of controlling arobotic arm, comprising: activating at least one independentlycontrollable fiducial located on a robotic arm, each of said fiducialsbeing a self-illuminating “active” fiducial such that each emits its ownlight when activated; determining by a processor the spatial location inthree dimensions of at least one fiducial; determining by the processorthe spatial location in three dimensions of a target object;continuously comparing the spatial locations of said at least one activefiducial and said target object; and moving said arm based on saidcontinuous comparing so as to reduce the difference between the spatiallocations of said at least one active fiducial and said target object,such that said at least one active fiducial is moved towards said targetobject.
 27. The method of claim 26, further comprising illuminating saidtarget object.